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Too many youth leave high school without the literacy skills that colleges and employers 
demand. In the Oakland Unified School District (OUSD), about half of secondary 
students read multiple years below their grade level. Despite many promising programs 


designed to increase literacy among younger students, schools struggle with finding effective 


ways for accelerating older, struggling readers. In search of a solution, OUSD began 


piloting Leveled Literacy Intervention (LLI), an intensive reading program, in secondary 
schools in 2015. Many school districts across the country have used LLI, which has shown 
promise in improving outcomes for students in early elementary grades. This research brief 
summarizes findings on the implementation and impacts of LLI in Oakland, where the 
district conducted the nation's first randomized controlled trial of LLI in secondary grades. 


BACKGROUND 


According to the Program for the International 
Assessment of Adult Competencies, 14 percent of 
American youth and young adults possess limited 
literacy skills. In OUSD, 52 percent of secondary 
students scored multiple years below their expected 
reading level on the Scholastic Reading Inventory 
in fall 2016, with almost one-third reading four 
or more years below grade level. Because literacy 
is associated with greater job opportunities and 
higher incomes, the stakes are high for finding 
effective ways to accelerate these students. 
In response, OUSD invested in piloting LLI in 
secondary schools and partnered with researchers 
at Mathematica to study the program's effective- 
ness with struggling readers in grades 6 to 9. 
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THE INTERVENTION 


LLI is a short-term, intensive intervention 
system designed to help teachers provide daily, 
small-group instruction to students who are not 
achieving grade-level expectations in reading. 

It is intended to supplement, rather than 
replace, regular literacy instruction. Although it 
was originally developed for students in early 
elementary grades, it has since expanded across 
K-12. Materials include a series of leveled texts 
and lesson guides of progressing difficulty that 
match to students’ reading level. Lessons focus 
on increasing reading volume, deepening and 
expanding comprehension, and promoting 
language development. 


Teachers begin by assessing students using a 
one-on-one assessment that matches students’ 
instructional and independent reading abilities 
to the text-level gradient used by LLI. Teachers 
then form small groups of three to five students 
with similar assessment scores and deliver 30- 
to 45-minute daily lessons. The recommended 
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ABOUT THE STUDY 
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program length ranges from 12 weeks to 24 or 
more weeks and depends on students’ starting 
reading level. For the starting reading levels of 
most students in this study (corresponding to 
grades 3 to 5), LLI recommends that students 
participate for 18 to 24 weeks. 


Past studies indicate that LLI is effective at 
improving the reading skills of younger students. 
In randomized controlled trials in diverse settings 
with students in kindergarten through grade 2, 
LLI produced significant gains in reading after 
12 to 18 weeks (Ransford-Kaldon et al. 2010; 
Ransford-Kaldon et al. 2013). A review of this 
research by the What Works Clearinghouse 
(2017) determined that LLI had positive effects 
on general reading achievement and potentially 
positive effects on reading fluency for beginning 
readers. In Oakland, elementary schools have 
offered LLI to students in early grades since 2012 
and have reported promising results. 


FINDINGS 


Students experienced different levels of LLI 
duration and intensity, and most fell short of the 
recommended minimum number of sessions. 
The average student assigned to LLI participated 
in the program for 16 weeks, although the duration 
varied significantly across students. School-level 
factors, such as the program start and end date, and 
individual student factors, such as the decision to 
stop participating in LLI, affected how long 
students were enrolled in the program. In addition, 
very few students received the recommended inten- 
sity of 4 or 5 days per week, as student attendance 
was inconsistent—on average, students assigned to 
LLI attended 2.1 out of 3.8 sessions offered per 
week—and only 2 of the 10 study schools offered 


daily sessions. As a result, 63 percent of students 
assigned to LLI received fewer than the total 
recommended minimum number of sessions. 


High schools particularly struggled with student 
participation and engagement. High school 
students assigned to LLI were 3.6 times less likely 
than students in grades 6 to 8 to actually receive 
LLI. In most cases, this disparity was because 
high school students (or their families) declined 
to participate in the program. High schoolers also 
attended significantly fewer sessions than students 
in middle grades. Teachers at each of the three high 
schools in the study reported challenges with 
student attendance or engagement, particularly 
with groups pulled out of other classes. For 
example, one teacher reported that students 
resented leaving a creative writing class they liked 
for LLI. Another teacher had to terminate the 
program in early February because of chronically 
low student attendance despite initial plans to 
offer LLI through the end of the school year. 


Program start and end dates varied widely across 
schools. Although most schools planned to 
start offering LLI in the fall, actual start dates 
ranged from October to February. Schools 
reported that securing the right LLI materials, 
receiving teacher training, and completing the 
initial one-on-one student assessments sometimes 
took longer than anticipated. End dates also 
varied from February to June. In some schools, 
the duration was fixed (for example, all of second 
semester), while in other schools, LLI was offered 
through the end of the school year and the dura- 
tion depended on individual student progress, 
though exit criteria varied or were not always 
specified. On average, students began the program 
in early December and ended in mid-April. 


Program duration and intensity for students assigned to LLI 


Weeks of LLI 
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Average: 16 weeks 


LLI sessions per week 


32 
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Share of students assigned to LLI (%) 


Average: 2.1 sessions per week 


Source: Mathematica calculations based on student data provided by OUSD. 


LLI Lesson Components 
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assessment 


Scheduling LLI classes proved difficult for 
some schools, but pulling students out of other 
classes presented other challenges. Of the 

10 study schools, 5 offered LLI in a scheduled 
class, 4 pulled students out of other classes, and 
1 used both models. Some schools noted that 
it was difficult to create a regularly scheduled LLI 
class because LLI serves a small number of students 
and their participation is not determined until 
after the start of the school year. Although offering 
LLI in pullout groups was logistically easier, 
students had to miss other classes and make up 
the work, which concerned some students, parents, 
and classroom teachers. Some LLI teachers tried 
to minimize this burden by holding sessions at 
a different time each day, but students in pullout 
groups were more likely to refuse LLI and attend 
fewer sessions than those in regularly scheduled 
classes, particularly at the high school level. 


Instructional fidelity was relatively low, 
primarily because of skipped or modified lesson 
components. A typical LLI lesson is fast-paced 
and has multiple components, with most being 5 
to 10 minutes long. Many teachers struggled to 
complete all of the components in a day’s lesson 
within the time allotted and either skipped or 
completed some in the following session. The most 
commonly skipped components were phonics/ 
word study and reading and assessment. Although 
teachers sometimes skipped phonics for timing 
reasons, some felt the content was not appropriate 
for the needs of their older students. The assess- 
ment component presented different challenges. 
In addition to time constraints, some teachers 
reported struggling to keep the rest of the group on 
task during one-on-one assessments. Conducting 
these assessments also required experience or 


training, familiarity with LLI’s online resources, 
and time to download and print the forms in 
advance. These requirements potentially presented 
barriers to teachers new to LLI and even to 
experienced teachers with limited planning time. 
Finally, it was common for teachers to modify 
the lesson guides in other ways, such as reorga- 
nizing the components or deviating from the 
suggested language. 


Although most teachers were new to LLI, not 
all of them received sufficient training in the 
program. Of the 20 teachers in the study, 14 
received training, typically in a two-day session led 
by an LLI representative, a half-day training led 
by OUSD central staff, or both. Only 1 of the 6 
teachers who did not receive training had previous 
experience with LLI (overall, 20 percent of 
teachers had previous experience with the program). 
‘The fidelity data suggest that these training 
opportunities might not have sufficiently prepared 
all teachers to conduct LLI as designed. During 
classroom observations, some teachers were found 
to need reminders about lesson timing, suggested 
modifications for shorter sessions, how to find and 
use the online resources to prepare lessons in advance, 
how to regularly assess students, and at times, how to 
conduct the lesson components as written. 


LLI had no impact on students’ reading 
comprehension and a negative impact on their 
mastery of English language arts (ELA)/ 
literacy standards. At the end of the school year, 
students in all grades took the Scholastic Reading 
Inventory, which assesses reading comprehension, 
and students in grades 6 to 8 took the Smarter 
Balanced Assessment, which tests mastery 
of grade-level standards in ELA/literacy. 


Student test scores before and after being assigned to LLI 
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o 

5 50 

g 

@ 40 

2 

m 30) * 

2 

% 20 

o 

@ 10 

2 0 

ee Spring 2016 Spring 2017 
(pre) (post) 


—— Control —@— Assigned to LLI 


* The effect is statistically significant at the 5 percent level. 


Source: Mathematica calculations based on student data provided by OUSD. 

Notes: Percentile scores range from 1 to 100 and are relative to all secondary students in OUSD. Control group averages 
were adjusted using ordinary least square regressions that account for small, random differences in pre-test scores and 
demographic characteristics between students assigned to LLI and students in the control group. 
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Compared with the control group, LLI students 
Scholastic Reading Inventory scores grew about 
the same between the fall and spring on average. 
And although students in the control group on 
average improved their Smarter Balanced ELA/ 
literacy scores from the previous year, LLI 
students performed slightly worse. Based on the 
typical annual growth in reading of students in 
these grades, the negative impact of being assigned 
to LLI on ELA/literacy scores is roughly 
equivalent to losing 5.5 months of learning. 


Students who received more LLI or were in 
pullout groups were particularly negatively 
affected, possibly as a result of missing grade- 
level content. Many students assigned to LLI 
received fewer than the minimum recommended 
number of sessions, which could affect the program’s 
effectiveness. However, receiving at least 48 
sessions also had no impact on reading compre- 
hension and had an even larger, negative impact 
on ELA/literacy scores (equivalent to losing about 
10 months of learning). In addition, the impact of 
LLI on student performance on both assessments 
was worse for students taught in pullout groups. 
As very few students in the control group received 
additional literacy supports beyond regular 
classroom instruction, these findings suggest that 
students in LLI might have been hurt by missing 
grade-level content. 


LOOKING AHEAD 


One reason OUSD elected to implement LLI in 
secondary schools is that—consistent with the 
experience of many of its elementary schools— 
past studies suggest the program is effective with 
students in early grades. However, aside from 
focusing on a very different grade span, these 
studies featured other key distinctions: students 
received the recommended number of sessions, 
fidelity ratings were high, and all teachers 
received eight days of professional development. 
Because of the relatively inconsistent imple- 
mentation of the program in OUSD secondary 
schools, it is difficult to pinpoint why LLI was 
not effective in this context. It is possible that 
after one or two more years, secondary schools 
and teachers would gain enough experience to 
overcome the early challenges they faced and 
implement the program to fidelity, leading to 
better results. 
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However, secondary schools and students might 
also have unique needs that are not well suited 
to LLI. For example, although there appear 

to be bigger downsides to pulling secondary 
students out of grade-level instruction to receive 
LLI, scheduling a regular LLI class can present 
logistical challenges for middle and high schools. 
In addition, because of students’ low starting 
reading levels, most schools used LLI materials 
designed for students in elementary grades— 
which might help explain why teachers skipped 
some lesson components and why high school 
students were less engaged. This study’s findings 
highlight the importance of assessing whether 
the success of instructional programs in one 
context can be replicated with different popula- 
tions and conditions. 

For more information, please contact 
Naihobe Gonzalez at ngonzalez@ 
mathematica-mpr.com 
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